Embodiments of the invention relate to data structure processing, in particular, for data structure processing for in-place updates with concurrent reads in a decomposed store for consistency of data.
There is an increasing trend towards doing business intelligence (BI) queries on real-time data in databases or tabled data. Traditionally, there is a strict separation between BI systems and online transaction processing (OLTP) systems. There is increasing market pressure for operational BI, and for both transactions and analytics to be performed on the same database or other forms of multithreaded applications.
In BI, analytics are performed on the same data where updates are happening. Conventionally, a big challenge with this is contention, between reads (queries) and updates. To avoid this contention traditionally, analytics are performed on a separate system from the updates, but this means the analytics ares always somewhat out of date. This contention problem has become worse in recent years, because data format has switched to column stores. Each object (record) is stored in a decomposed fashion, with different parts on different pages. To deal with the contention, readers and updaters have to simultaneously latch the entire record—either all pages, or other record-level enclosures (e.g., record id ranges). Without this latch, the updater updates a record in decomposed fashion, and the reader potentially sees an inconsistent version of the record. With the latch, readers have to pay for an expensive latch per record overhead, which is especially problematic for BI queries that often scan large portions of the entire data set. Updaters have to wait for all the readers to ‘drain out’ before they can access a record.